Classifying images in overlapping groups of images using convolutional neural networks

ABSTRACT

The present disclosure relates to training a machine learning model to classify images. An example method generally includes receiving a training data set including images in a first category and images in a second category. A convolutional neural network (CNN) is trained using the training data set, and a feature map is generated from layers of the CNN based on features of images in the training data set. A first area in the feature map including images in the first category and a second area in the feature map where images in the first category overlap with images in the second category are identified. The first category is split into a first subcategory corresponding to the first area and a second subcategory corresponding to the second area. The CNN is retrained based on the images in the first subcategory, images in the second subcategory, and images in the second category.

BACKGROUND

The present disclosure relates to image classification, and morespecifically, to classifying images in overlapping groups of images.

Image recognition is a task that may be performed in variousenvironments. For example, in industrial environments, image recognitionmay be used to identify anomalies in manufactured goods. For any givenitem being manufactured, anomalies may range from minor anomalies thatdo not substantially affect the functionality of the item (e.g., arecosmetic or other minor manufacturing errors) to major anomalies thatrender the item unusable or unsellable (e.g., stuck pixels on an LCDpanel, malformed memory cells on solid state storage, malformed logicalunits on a processor, etc.).

Because of the variety of anomalies that may be generated during themanufacturing of various goods, anomalies may further be classified intocategories that define specific types of anomalies. Generally, imagesrepresenting a specific type of anomaly may have similar appearances,and machine learning models such as convolutional neural networks may betrained to recognize these types of anomalies. However, in some cases,images representing a specific type of anomaly may have a distributionsuch that some of the images representing that specific type of anomalyactually have an appearance similar to images representing a differenttype of anomaly.

Further, anomalies may arise during the manufacturing process atdifferent rates. Some anomalies may be relatively common, while otheranomalies may be less common. A training data set that includesanomalies that are detected over an amount of time may, thus, include alarge number of images that correspond to relatively common anomaliesand a significantly smaller number of images that correspond to lesscommon anomalies. Further, the number of images of an anomaly thatappears similar to other anomalies may be relatively small. Training amachine learning model, such as a convolutional neural network, based onsuch a training data set may result in poor performance, as the modelmay have insufficient training data for recognizing less commonanomalies or anomalies that appear to be similar to other classes ofanomalies.

SUMMARY

One embodiment disclosed herein includes a method for training a machinelearning model. The method generally includes receiving a training dataset including images in a first category and images in a secondcategory. A convolutional neural network (CNN) is trained using thereceived training data set. A feature map is generated from one or morelayers of the CNN based on one or more features of images in thetraining data set. A first area in the feature map is identified, wherethe first area includes images in the first category, and a second areain the feature map is identified, where images in the first categoryoverlap with images in the second category. Based on the identifiedfirst and second areas, the first category of images is split into afirst subcategory corresponding to the first area in the feature map anda second subcategory corresponding to the second area in the featuremap. The CNN is retrained based on images in the first subcategory,images in the second subcategory, and images in the second category.

Another embodiment includes a processor and a memory having instructionsstored thereon which, when executed on the processor, performs anoperation for training a machine learning model. The operation generallyincludes receiving a training data set including images in a firstcategory and images in a second category. A convolutional neural network(CNN) is trained using the received training data set. A feature map isgenerated from one or more layers of the CNN based on one or morefeatures of images in the training data set. A first area in the featuremap is identified, where the first area includes images in the firstcategory, and a second area in the feature map is identified, whereimages in the first category overlap with images in the second category.Based on the identified first and second areas, the first category ofimages is split into a first subcategory corresponding to the first areain the feature map and a second subcategory corresponding to the secondarea in the feature map. The CNN is retrained based on images in thefirst subcategory, images in the second subcategory, and images in thesecond category.

Still another embodiment includes a computer-readable storage mediumhaving instructions stored thereon, which, when executed on a processor,performs an operation for training a machine learning model. Theoperation generally includes receiving a training data set includingimages in a first category and images in a second category. Aconvolutional neural network (CNN) is trained using the receivedtraining data set. A feature map is generated from one or more layers ofthe CNN based on one or more features of images in the training dataset. A first area in the feature map is identified, where the first areaincludes images in the first category, and a second area in the featuremap is identified, where images in the first category overlap withimages in the second category. Based on the identified first and secondareas, the first category of images is split into a first subcategorycorresponding to the first area in the feature map and a secondsubcategory corresponding to the second area in the feature map. The CNNis retrained based on images in the first subcategory, images in thesecond subcategory, and images in the second category.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 illustrates an example networked environment in which a machinelearning model is trained using image data from a first category ofimages that overlaps with image data from a second category of images,according to one embodiment.

FIG. 2 illustrates example operations for training a machine learningmodel using a training data set of images from a first category ofimages that overlaps with images from a second category of images,according to one embodiment.

FIG. 3 illustrates example operations for classifying images using amachine learning model trained using a data set of images tagged withinternal categories mapped to user-defined categories, according to oneembodiment.

FIG. 4 illustrates an example feature map and associated mapping ofconvolutional neural network (CNN) categories to user-definedcategories, according to one embodiment.

FIG. 5 illustrates an example system in which aspects of the presentdisclosure may be performed.

DETAILED DESCRIPTION

Embodiments presented herein describe techniques for training a machinelearning model to classify images based on training data sets of imagesin a first category of images that potentially includes images that aresimilar to those in a second category of images. Generally, an initialmodel may be trained using a training data set of pre-classified images.After training the initial model, a feature may be extracted from alayer of the initial model, and a feature map may be generated based onthe extracted feature to identify clusters in the feature map thatdifferent types of images belong to. If the feature map results in aclustering for a first class of images that includes some overlap with aclustering for a second class of images, the first class of images maybe split into a first subclass of the non-overlapping images and asecond subclass of the overlapping images, and the model may beretrained using the first subclass of images, the second subclass ofimages, and the second class of images.

By using feature maps and identification of overlapping clusters in afeature map to retrain a machine learning model, embodiments presentedherein may generate machine learning models that can accurately classifyimages received for analysis. For images that include features that arecommon to a plurality of categories, a machine learning model trainedusing feature maps and identification of overlapping clusters in afeature map may classify such images into an internal category that mapsto a user-defined category. Instead of classifying an image based on acoarser categorization that may result in the incorrect classificationof an image, embodiments presented herein may classify an image into thecorrect user-defined category based on classifications of images intosmaller sub-categories.

FIG. 1 illustrates an example networked computing environment in which amachine learning model training system trains a model for classifyingimages based on a training data set of images in a first category ofimages that potentially includes images that are similar to those in asecond category of images, according to an embodiment of the presentdisclosure. As illustrated, computing environment 100 includes a clientdevice 120, an application server 130, a model generator 140, and atraining data store 150.

Client device 120 generally is representative of a computing device onwhich a user can initiate a request to classify an image or set ofimages by an application executing on application server 130. Clientdevice 120 may be, for example, a laptop computer, a desktop computer, athin client, a tablet computer, a mobile computing device, and the like.As illustrated, client device 120 includes a user interface 122. Userinterface 122 allows a user of client device 120 to interact with anapplication that uses a machine learning model generated by modelgenerator 140. In some embodiments, user interface 122 may display atext editing interface or other graphical interface that allows a userto specify images to classify using application server 130 and mayexpose a file upload mechanism or other batch data processing mechanismsfor uploading images to application server 130 for analysis.

While client device 120 as illustrated includes a user interface throughwhich a user can specify images to be classified using applicationserver 130, it should be recognized that client device 120 need notinclude a user interface to interact with application server 130. Forexample, various Internet of Things (IoT) or other connected devices,such as industrial robots, process control systems, or the like may beconnected to application server 130 to upload images, in real time, foranalysis.

In some embodiments, client device 120 may take one or more actionsbased on the classification of an image received from application server130. For example, in a manufacturing or quality control system, clientdevice 120 may use the classification of images to determine whether toallow a manufactured item to proceed to the next manufacturing step,send the manufactured item to a remediation step, or discard themanufactured item. In another example, in a semiconductor manufacturingexample, client device 120 can designate dies fabricated on asemiconductor wafer for different bins of products, such as processorswith different performance characteristics, based on the classificationof images captured for each die on the semiconductor wafer. It should berecognized, however, that a variety of actions may be taken by clientdevice 120 based on the classification of images based on the specificenvironment in which client device 120 is deployed.

Application server 130 generally receives and processes requests toclassify images received from a client device 120 based on a machinelearning model trained by model generator 140. As illustrated,application server 130 includes an image data receiver 132 and an imageclassifier 134. While application server 130 is illustrated as includingimage data receiver 132 and image classifier 134, it should berecognized that these components need not reside on the same system.

Image data receiver 132 generally provides an image upload interfacethrough which images are received from client device 120. Generally,image data receiver 132 may be configured to receive images from aclient device 120 in various file formats, such as the JointPhotographic Experts Group (JPEG) format, Tagged Image File Format(TIFF), Portable Network Graphics (PNG) format, or other image fileformats. In some embodiments, image data receiver 132 may be configuredto receive images in one or more supported file formats and convert thereceived images into a file format usable by image classifier 134 forclassifying the received images. In some embodiments, image datareceiver 132 may be configured to resize received images into a standardresolution prior to passing the received images to image classifier 134for analysis. In some embodiments, image data receiver 132 may beconfigured to perform other postprocessing operations on the receivedimages to improve the quality of the received images. These operationsmay include, for example, noise reduction, exposure compensation,sharpening, and other post-processing operations that may improve thequality of the received images. In some embodiments, image data receiver132 may additionally determine whether an image is of sufficient qualityto proceed with classification and may reject an image and/or request areplacement image if image data receiver 132 determines that the imageis of insufficient quality to proceed with classification.

Image classifier 134 generally uses a machine learning model, such as aconvolutional neural network (CNN), to classify images into a category.As discussed in further detail below, the machine learning model used byimage classifier 134 may be trained using sub-categories of images in auser-defined category such that the machine learning model classifies areceived image as one of the sub-categories of images. Generally, imageclassifier 134 may receive, from model generator 140, a definition of amachine learning model trained using a plurality of internal categories,which may include sub-categories of images in a user-defined category,and a mapping of each of the plurality of internal categories in themachine learning model to a user-defined category of images in atraining data set used to initially train the machine learning model.

When image classifier 134 receives an image for analysis from a clientdevice 120 through image data receiver 132, image classifier 134 mayclassify the image into one of the internal categories in the machinelearning model. After classifying the image into one of the internalcategories, image classifier 134 can examine the mapping of the internalcategories to user-defined categories to identify that user-definedcategory that an image is to be classified in. Image classifier 134 canreturn the user-defined category to client device 120 as theclassification of the received image based on the mapping of theinternal categories to the user-defined categories.

Model generator 140 generally trains a machine learning model toclassify images based on a training data set of images from trainingdata stores 150. Generally, model generator 140 uses supervised learningtechniques to train a machine learning model, and each image in atraining data set used to train the machine learning model may be taggedor otherwise associated with a user-defined category. The machinelearning model generated by model generator 140 may, for example, be aconvolutional neural network, which is a neural network that includes aplurality of convolutional layers that assigns weights to differentfeatures in an input image and classifies the input image based on theassigned weights in each of the layers. However, it should be recognizedthat model generator 140 may be configured to train other types ofmachine learning models that may be used in image recognition, such asother deep neural networks, Bayes classifiers, decision trees, or othermachine learning models that can be trained using supervised learningtechniques. As illustrated, model generator 140 includes a feature mapgenerator 142, a category splitter 144, and a model retrainer 146.

Feature map generator 142 generally is configured to generate an initialmachine learning model using a training data set of user-provided andtagged data in training data repository 150. In some embodiments,feature map generator 142 may pre-process the training data set togenerate a balanced training data set such that each user-definedcategory includes a similar number of images for use in training theinitial machine learning model (e.g., such that the difference betweenthe number of images in a first user-defined category and the number ofimages in a second user-defined category is within a threshold amount).By pre-processing the training data set to generate the balancedtraining data set, feature map generator 142 can generate a trainingdata set that does not bias classification towards a category that hassignificantly more images in the training data set than categories thathave fewer images in the training data set. For example, given atraining data set with 15,000 images in a first category and 3,000images in a second category, images in the second category have a higherlikelihood of being classified in the first category because of theunbalance in the training data set. Balancing the training data set sothat there are a similar number of images in the first and secondcategories may avoid biasing a machine learning model towards onecategory of images or other data in the training data set.

After balancing the training data set, feature map generator trains aninitial machine learning model. The initial machine learning model maybe trained such that a feature map may be generated for each of aplurality of features identified in the images that may be used todetermine whether an image is in a specific category. In someembodiments, where the machine learning model is a convolutional neuralnetwork or other type of neural network, each feature may correspond toa hidden layer of a neural network (i.e., a layer between an input layerof the machine learning model and an output layer of the machinelearning model). To generate a feature map, feature map generator canperform an aggregation operation over a feature from a layer of amachine learning model. The aggregation operation may use, for example,k-nearest neighbor (kNN) techniques or other distance-based techniquesto generate a feature map showing the locations, on the feature map, ofeach image in the training data set for a specific selected feature.

Feature map generator 142 generally passes the generated feature map tocategory splitter 144, which uses the feature map to identifyuser-defined categories that can be split into a plurality ofsub-categories. Generally, category splitter 144 can identify auser-defined category that can be split into a plurality ofsub-categories by identifying, in the feature map, areas in which imagesbelonging to the user-defined category overlap with images belonging toa different user-defined category. In some embodiments, the images inthe area of the generated feature map that overlaps with images belongto another user-defined category may be designated as a sub-category ofimages if these images are determined to be in a cluster that is athreshold distance away from the center or some other pre-defined pointon the feature map for the user-define category of images.

When category splitter 144 identifies a user-defined category of imagesA that can be split into a plurality of sub-categories {A, A′, A″, . . .}, category splitter 144 can generate a new training data set of datafrom the training data set used by feature map generator 142 to generatethe initial machine learning model. The user-defined category of imagesA may overlap with a single user-defined category of images B such that{A′, A″, . . . } each overlap with a portion of the feature mapincluding images from user-defined category B. The user-defined categoryof images A may, in some embodiments, overlap with multiple otheruser-defined categories of images {B, C, . . . } such that A′ overlapswith the portion of the feature map including images from user-definedcategory B, A″ overlaps with the portion of the feature map includingimages from user-defined category C, and so on. In still furtherscenarios, a portion of the feature map including images fromuser-defined category of images A may overlap with a portion of thefeature map including images from user-defined category of images B(designated the A-B overlapping area), and a portion of the feature mapincluding images from user-defined category of images C may overlap withthe A-B overlapping area. In such a scenario, each of user-definedcategories A, B, and C may be split into sub-categories A and A′, B andB′, and C and C′, respectively, and A′, B′, and C′ may include images inthe A-B overlapping area of the feature map from their respective parentuser-defined categories.

The new training data set may include the plurality of sub-categories,with one of the sub-categories retaining the user-defined category nameand the other sub-categories being designated with different names(e.g., with one or more characters prepended to the user-definedcategory name or appended to the user-defined category name). Imagesthat are not in the area of the feature map overlapping with images froma second user-defined category may remain tagged with the user-definedcategory name. Category splitter 144 can rewrite the tags of images thatare within an area of the feature map overlapping with images from asecond user-defined category with a sub-category name that is differentfrom the user-defined category name.

Further, after category splitter 144 determines that a user-definedcategory can be divided into a plurality of sub-categories, categorysplitter 144 may generate a mapping between each of the plurality ofsub-categories and the user-defined category. The mapping may begenerated, for example, in a mapping table or other lookup table that isprovided to application server 130 with the retrained machine learningmodel generated by model retrainer 146, as discussed above. Generally, amapping may include a listing of the categories of images or other datagenerated by category splitter 144 and the corresponding user-definedcategory for each category generated by category splitter 144.

In some embodiments, category splitter 144 can rebalance the newtraining data set prior to providing the new training data set to modelretrainer 146 for use in retraining the machine learning model.Generally, as discussed above, rebalancing the new training data set mayinclude reducing the size of the training data set such that eachcategory of data in the new training data set includes a similar numberof images. That is, for a new training data set {A, A′, B, B′, C}, whereimages in sub-categories A and A′ belong to user-defined category A andimages in sub-categories B and B′ belong to user-defined category B,each of A, A′, B, B′, and C may have a similar number of images for usein retraining the model.

In some embodiments, category splitter 144 can determine that asub-category of a first user-defined category overlaps with asub-category of a second user-defined category in the training data set.That is, given a first user-defined category A and a second user-definedcategory B, sub-categories A′ and B′ overlap in the feature map. In someembodiments, where sub-categories A′ and B′ overlap in the feature map,category splitter 144 can determine that other features should be usedto split categories of data based on an aggregated feature map. In someembodiments, other aggregation parameters may be used to determinewhether to split a user-defined category into a plurality ofsub-categories. For example, the aggregation parameters may be changedto use a different minimum and maximum number of neighbors in theaggregation, a different similarity metric, or the like. In someembodiments, features from a different layer in the machine learningmodel may be used in conjunction with the feature used to generate thefeature map to generate a new feature map.

Model retrainer 146 uses the new training data set generated by categorysplitter 144 to retrain the machine learning model initially trained byfeature map generator 142. Generally, by retraining the machine learningmodel using the new training data set generated by category splitter144, model retrainer 146 may generate a machine learning model that istrained to categorize received images into one of a plurality ofinternal categories, which may include sub-categories of data generatedfrom identifying overlapping areas in a feature map, as discussed above,and user-defined categories of data that do not include areas in thefeature map that overlap with areas in the feature map associated withother categories.

In some embodiments, model retrainer 146 may additionally retrain amachine learning model based on new training data. To retrain a machinelearning model based on new training data, model retrainer 146 canexamine the feature on which category splitter 144 generated a featuremap to identify the internal category to which the new training databelongs. Model retrainer 146 can further rebalance the training dataset, augmented by the new training data, to ensure that the model isretrained on a training data set having substantially similar numbers ofimages in each internal category of images.

Training data store 150 generally is representative of a data store thatcan store a training data set used by model generator 140 to train amachine learning model for classifying images, as discussed above.Training data store 150 may be a relational database, a non-relationaldatabase, a flat file store, or other appropriate data store in whichimages and the associated user-defined category for each image can bestored.

FIG. 2 illustrates example operations that may be performed by a systemto train a machine learning model based on a training data set includingimages or other content in a first category that overlaps with images orother content in a second category, according to an embodiment.

As illustrated, operations 200 begin at block 210, where the systemreceives a training data set including images in a first category andimages in a second category. As discussed, the training data setgenerally includes a plurality of images, and each image in theplurality of images may be labeled or otherwise associated with auser-defined category.

At block 220, the system generates a feature map from one or more layersof a trained convolutional neural network (CNN) based on one or morefeatures of images in the training data set. To generate the featuremap, the system may train an initial CNN based on the received trainingdata set. The system may then select a first feature, associated withone of the layers of the initial CNN, on which to generate a feature mapusing aggregation methods, such as k-nearest neighbor. Generally, thefeature map may be generated with a plurality of clusters, with eachcluster in the feature map representing a grouping of similar images. Insome cases, a cluster may include images belonging to a firstuser-defined category and images belonging to a second user-definedcategory.

At block 230, the system identifies a first area in the feature mapincluding images in the first category and a second area in the featuremap where images in the first category overlap with images in the secondcategory. Generally, as discussed, the first area in the feature map maybe an area representing images having features that do not overlap withfeatures of images in a different user-defined category, while thesecond area may be an area in the feature map representing images havingfeatures that overlap with features of images in a differentuser-defined category. In some embodiments, the second area may bedefined as an area of images belonging to a first category that is athreshold distance away from the first area in the feature map. In someembodiments, if the system determines that the second area in thefeature map overlaps with a similar second area in the feature map forimages belonging to the second user-defined category that is separatefrom a primary area in the feature map for images belonging to thesecond user-defined category, the system can identify a differentfeature to use for generating the feature maps, alone or in combinationwith the currently selected feature, and can adjust parameters used toaggregate the images into different areas of the feature map.

At block 240, based on the identified first and second areas, the systemsplits the first category of images into a first subcategorycorresponding to the first area in the feature map and a secondsubcategory corresponding to the second area in the feature map. Insplitting the first category of images into a first subcategory and asecond subcategory, the system can generate a new training data set withimages in the first area in the feature map retaining a tag specifyingthe corresponding user-defined category and images in the second area inthe feature map being tagged with a different label. This differentlabel may be derived from the user-defined category. Additionally, basedon the label generated for the images in the second area, the system cangenerate a mapping between internal categories used in retraining theconvolutional neural network (or other machine learning model) and theuser-defined categories originally included in the training data set sothat an output of an image classifier using the retrained convolutionalneural network corresponds to one of the user-defined categories ratherthan an internal category.

At block 250, the system retrains the CNN based on a training data setincluding images in the first subcategory, images in the secondsubcategory, and images in the second user-defined category of images.By retraining the CNN based on a training data set including the imagesin the first subcategory, images in the second subcategory, and imagesin the second user-defined category of images, the CNN may be trained toclassify a received image as being in the first subcategory, the secondsubcategory (i.e., the subcategory of images that are mapped to an areain the feature map overlapping with the area in the feature map to whichimages from the second user-defined category are mapped). Arepresentation of the retrained CNN may be saved to one or more filesthat can be used by an application to execute the retrained CNN in astandalone application or may otherwise be deployed to an applicationserver, such as application server 130 illustrated in FIG. 1, forexecution.

FIG. 3 illustrates example operations for classifying images using amachine learning model trained using a data set of images tagged withinternal categories mapped to user-defined categories, according to anembodiment.

As illustrated, operations 300 begin at block 310, where a systemreceives an image to classify using a convolutional neural network(CNN). The image to classify may be received as a user upload from aclient device as a single image or an image in a batch of images foranalysis. In some embodiments, the image to classify may be received inreal time from a client device, such as a monitoring computer or processcontroller in an industrial environment.

At block 320, the system classifies the image into one of a plurality ofinternal categories used by the CNN to classify images. The internalcategories generally correspond to the categories with which images usedto train the CNN are tagged. As discussed above, the internal categoriesmay include subcategories generated from a feature map for images in auser-defined category and user-defined categories that are not splitinto subcategories.

At block 330, the system examines a mapping between the one of theplurality of internal categories to which the image is classified andthe user-defined image categories. As discussed, the mapping may beincluded in a table generated during training of the CNN and mayidentify, for each internal category, the corresponding user-definedimage category to which the internal category is mapped.

At block 340, the system outputs, as the classification of the image,the user-defined image category to which the one of the plurality ofinternal categories is mapped.

FIG. 4 illustrates an example feature map generated during training of aconvolutional neural network and a mapping of internal categoriesgenerated from the feature map to user-defined categories, according toan embodiment.

As illustrated, feature map 400 includes a mapping of two user-definedcategories of images. User-defined category A may be represented by thestars on the feature map 400, and user-defined category B may berepresented by the diamonds on the feature map 400. Each item on thefeature map 400 (e.g., each star or diamond) may be placed at a locationon the feature map 400 based on a feature value for each image generatedby the convolutional neural network. As discussed, the feature valuesmay be placed on the map using various techniques, such as k-nearestneighbor or other aggregation techniques.

As illustrated, images in user-defined category A are mapped to twospaces in the feature map 400. A first space, designated as featurespace A, corresponds to the images in user-defined category A that havefeatures with values that do not overlap with the feature values forimages in user-defined category B. A second space, designated as featurespace A′, corresponds to the images in user-defined category A that havefeatures with values that overlap with the feature values for images inuser-defined category B. Finally, as illustrated, the images inuser-defined category B are mapped to an area in feature map 400designated as feature space B.

Based on the identifying that some images in user-defined category A aremapped to the same area in feature space 400 as images in user-definedcategory B, a system can determine that user-defined category A shouldbe split into two internal categories, A and A′. Internal category Agenerally corresponds to the images in feature space A, as illustratedin FIG. 4. Internal category A′ generally corresponds to the images infeature space A′, which, as discussed, partially overlaps with featurespace B (i.e., the feature space to which images in user-definedcategory B are mapped).

Based on the determination that user-defined category A should be splitinto internal categories A and A′, the system can generate a newtraining data set or edit the current training data set to assign alabel corresponding to internal category A′ to the images havingfeatures mapped to feature space A′ in feature map 400. Additionally,the system can generate a map 410 between the internal categoriesgenerated by the system and used to retrain the convolutional neuralnetwork and the user-defined categories, which, as discussed, may beused after the convolutional neural network has categorized an image toone of the internal categories, to identify the user-defined category towhich the image belongs.

FIG. 5 illustrates an example application server 500 that trains amachine learning model (e.g., a convolutional neural network) torecognize images by splitting user-defined categories of images intointernal categories based on a feature map generated for images in atraining data set, according to an embodiment. As shown, applicationserver 500 includes, without limitation, a central processing unit 502,one or more I/O device interfaces 504, which may allow for theconnection of various I/O devices 514 (e.g., keyboards, displays, mousedevices, pen input, etc.) to the entity analytics system 500, networkinterface 506, a memory 508, storage 510, and an interconnect 512.

CPU 502 may retrieve and execute programming instructions stored in thememory 508. Similarly, the CPU 502 may retrieve and store applicationresiding in the memory 508. The interconnect 512 transmits programminginstructions and application data among the CPU 502, I/O deviceinterface 504, network interface 506, memory 508, and storage 510. CPU502 is included to be representative of a single CPU, multiple CPUs, asingle CPU having multiple processing cores, and the like. Additionally,the memory 508 is included to be representative of a random accessmemory. Furthermore, the storage 510 may be a disk drive. Although shownas a single unit, the storage 510 may be a combination of fixed and/orremovable storage devices, such as fixed disc drives, solid statedrives, removable memory cards or optical storage, network attachedstorage (NAS), or a storage area-network (SAN).

As illustrated, memory 508 includes an image data receiver 520, imageclassifier 530, feature map generator 540, category splitter 550, andmodel retrainer 560. Image data receiver 520 generally receives imagesfrom an external source (e.g., a client device) or a camera connected tosystem 500 via I/O interface 504 for analysis. Image data receiver 520may be configured to perform one or more postprocessing operations on areceived image prior to providing the image to image classifier 530 forclassification.

Image classifier 530 uses a machine learning model trained using atraining data set of images split into a plurality of internalcategories to classify an image received from image data receiver 520.After classifying an image to one of a plurality of internal categories,image classifier 530 can use a map between the internal categories anduser-defined categories to determine the user-defined category that animage should be classified in. Image classifier 530 may output thedetermined user-defined category, for example, for display in agraphical user interface on a display connected to system 500, to aremote system for display, or the like.

Feature map generator 540 generally receives or retrieves a trainingdata set of images from a remote data source or training data store 570in storage 110 and trains an initial machine learning model (e.g., aconvolutional neural network) using the retrieved training data set. Asdiscussed, the training data set may include a plurality of imagescategorized a priori to a plurality of categories. In some embodiments,feature map generator 540 can balance the number of images included ineach user-defined category of images prior to training the initialmachine learning model so that the initial machine learning model is notbiased towards categorizing received images to the user-defined categoryhaving a disproportionately large number of images in the training dataset. After training the initial machine learning model, feature mapgenerator 540 generally selects at least a first feature in the machinelearning model on which to generate a feature map and generates afeature map for the training data set of images.

Category splitter 550 uses the feature map generated by feature mapgenerator 540 to determine whether a user-defined category of imagesshould be split into a plurality of internal categories. Generally, auser-defined category of images may be split into a plurality ofinternal categories if a portion of the user-defined category of imagesis mapped to a first feature area in the feature map and a portion ofthe user-defined category of images is mapped to a second feature areain the feature map that overlaps with the feature area for a differentuser-defined category of images. When category splitter 550 determinesthat a user-defined category of images should be split, categorysplitter 550 can edit the labels associated with each image in theuser-defined category so that images mapped to the second feature areaof the feature map are labeled or tagged with a different internalcategorization from the user-defined category that these images wereoriginally labeled with.

Model retrainer 560 uses a training data set generated by categorysplitter 550 to generate a retrained machine learning model thatclassifies images or other data into one of a plurality of internalcategories identified by category splitter 550. Once retrained, themachine learning model may be deployed to image classifier 530 for usein classifying images initially into one of the internal categories andsubsequently into the appropriate user-defined category based on amapping between the internal categories and the user-defined categories,as discussed above.

Storage 510, as illustrated, includes a training data store 570.Training data store 570 generally provides a storage repository in whichimages used to train and retrain a machine learning model may be stored.Each of the images stored in training data store 570 may be labeled witha user-defined category, and the images and associated labels may beused by feature map generator 540, category splitter 550, and modelretrainer 560 to generate a machine learning model that classifiesimages into one of a plurality of internal categories, as discussedabove.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

In the following, reference is made to embodiments presented in thisdisclosure. However, the scope of the present disclosure is not limitedto specific described embodiments. Instead, any combination of thefollowing features and elements, whether related to differentembodiments or not, is contemplated to implement and practicecontemplated embodiments. Furthermore, although embodiments disclosedherein may achieve advantages over other possible solutions or over theprior art, whether or not a particular advantage is achieved by a givenembodiment is not limiting of the scope of the present disclosure. Thus,the following aspects, features, embodiments and advantages are merelyillustrative and are not considered elements or limitations of theappended claims except where explicitly recited in a claim(s). Likewise,reference to “the invention” shall not be construed as a generalizationof any inventive subject matter disclosed herein and shall not beconsidered to be an element or limitation of the appended claims exceptwhere explicitly recited in a claim(s).

Aspects of the present invention may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.”

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Embodiments of the invention may be provided to end users through acloud computing infrastructure. Cloud computing generally refers to theprovision of scalable computing resources as a service over a network.More formally, cloud computing may be defined as a computing capabilitythat provides an abstraction between the computing resource and itsunderlying technical architecture (e.g., servers, storage, networks),enabling convenient, on-demand network access to a shared pool ofconfigurable computing resources that can be rapidly provisioned andreleased with minimal management effort or service provider interaction.Thus, cloud computing allows a user to access virtual computingresources (e.g., storage, data, applications, and even completevirtualized computing systems) in “the cloud,” without regard for theunderlying physical systems (or locations of those systems) used toprovide the computing resources.

Typically, cloud computing resources are provided to a user on apay-per-use basis, where users are charged only for the computingresources actually used (e.g. an amount of storage space consumed by auser or a number of virtualized systems instantiated by the user). Auser can access any of the resources that reside in the cloud at anytime, and from anywhere across the Internet. In context of the presentinvention, a user may access applications or related data available inthe cloud. For example, the Application 132 could execute on a computingsystem in the cloud and perform read requests on a database based onindices included in an index data table and/or perform write requests ona database and maintain entries in the index data table forlow-frequency data. In such a case, the Application 132 could performdatabase transactions and store and maintain information in the indexdata tables at a storage location in the cloud. Doing so allows a userto access this information from any computing system attached to anetwork connected to the cloud (e.g., the Internet).

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed is:
 1. A method for training a machine learning model toclassify images, comprising: receiving a training data set includingimages in a first category and images in a second category; training aconvolutional neural network (CNN) using the received training data set;generating a feature map from one or more layers of the CNN based on oneor more features of images in the training data set; identifying a firstarea in the feature map including images in the first category and asecond area in the feature map where images in the first categoryoverlap with images in the second category; based on the identifiedfirst and second areas, splitting the first category of images into afirst subcategory corresponding to the first area in the feature map anda second subcategory corresponding to the second area in the featuremap; and retraining the CNN based at least in part on a training dataset including images in the first subcategory, images in the secondsubcategory, and images in the second category.
 2. The method of claim1, further comprising: deploying the retrained CNN to an imageclassifier, wherein the image classifier is configured to: receive animage for classification; classify the image using the retrained CNNinto one of a plurality of internal categories, the internal categoriesincluding at least the first subcategory, the second subcategory, andthe second category; identify a category for the image based on amapping between the plurality of internal categories and a plurality ofcategories; and return the identified category as the classification ofthe image.
 3. The method of claim 1, further comprising: identifying athird area in the feature map where images in a third category overlapwith images in the second subcategory; and generating a thirdsubcategory corresponding to the third area in the feature map, whereinthe CNN is retrained based further on images in the third subcategoryand images in the third category.
 4. The method of claim 1, whereintraining the CNN using the received training data set comprises:determining a number of images in each of the first category and thesecond category; and balancing images in the first category and thesecond category such that a difference between the number of images inthe first category and the second category is within a threshold amount.5. The method of claim 1, wherein splitting the first category of imagesinto the first subcategory and the second subcategory comprises:determining that the second area in the feature map overlaps with anarea in the feature map in which a subset of images in the secondcategory are mapped, the area in the feature map in which the subset ofimages in the second category are mapped being a threshold distance awayfrom an area in the feature map in which remaining images in the secondcategory are mapped; selecting a second feature on which to generate afeature map; and splitting the first category of images into the firstsubcategory and the second subcategory based on the feature mapgenerated for the selected second feature.
 6. The method of claim 1,wherein splitting the first category of images into the firstsubcategory and the second subcategory comprises: determining that thesecond area in the feature map overlaps with an area in the feature mapin which a subset of images in the second category are mapped, the areain the feature map in which the subset of images in the second categoryare mapped being a threshold distance away from an area in the featuremap in which remaining images in the second category are mapped;modifying one or more aggregation parameters used to generate thefeature map; regenerating the feature map for the one or more featuresbased on the modified aggregation parameters; and splitting the firstcategory of images into the first subcategory and the second subcategorybased on the regenerated feature map.
 7. The method of claim 1, whereinretraining the CNN based on a training data set including images in thefirst subcategory, images in the second subcategory, and images in thesecond category comprises: determining a number of images in each of thefirst subcategory, the second subcategory, and the second category; andbalancing images in the first subcategory, the second subcategory andthe second category such that a difference between the number of imagesin the first subcategory, the number of images in the secondsubcategory, and the second category is within a threshold amount. 8.The method of claim 1, further comprising: receiving a set of images toadd to the training data set; classifying the received set of imagesinto one of the first subcategory, the second subcategory, or the secondcategory based on a value of the one or more features for each image inthe received set of images to add to the training data set; andretraining the CNN based on the training data set and the received setof images to add to the training data set.
 9. A system, comprising: aprocessor; and a memory having instructions stored thereon which, whenexecuted by the processor, performs an operation for training a machinelearning model to classify images, the operation comprising: receiving atraining data set including images in a first category and images in asecond category; training a convolutional neural network (CNN) using thereceived training data set; generating a feature map from one or morelayers of the CNN based on one or more features of images in thetraining data set; identifying a first area in the feature map includingimages in the first category and a second area in the feature map whereimages in the first category overlap with images in the second category;based on the identified first and second areas, splitting the firstcategory of images into a first subcategory corresponding to the firstarea in the feature map and a second subcategory corresponding to thesecond area in the feature map; and retraining the CNN based on atraining data set including images in the first subcategory, images inthe second subcategory, and images in the second category.
 10. Thesystem of claim 9, wherein the operation further comprises: deployingthe retrained CNN to an image classifier, wherein the image classifieris configured to: receive an image for classification; classify theimage using the retrained CNN into one of a plurality of internalcategories, the internal categories including at least the firstsubcategory, the second subcategory, and the second category; identify acategory for the image based on a mapping between the plurality ofinternal categories and a plurality of categories; and return theidentified category as the classification of the image.
 11. The systemof claim 9, wherein the operation further comprises: identifying a thirdarea in the feature map where images in a third category overlap withimages in the second subcategory; and generating a third subcategorycorresponding to the third area in the feature map, wherein the CNN isretrained based further on images in the third subcategory and images inthe third category
 12. The system of claim 9, wherein training the CNNusing the received training data set comprises: determining a number ofimages in each of the first category and the second category; andbalancing images in the first category and the second category such thata difference between the number of images in the first category and thesecond category is within a threshold amount.
 13. The system of claim 9,wherein splitting the first category of images into the firstsubcategory and the second subcategory comprises: determining that thesecond area in the feature map overlaps with an area in the feature mapin which a subset of images in the second category are mapped, the areain the feature map in which the subset of images in the second categoryare mapped being a threshold distance away from an area in the featuremap in which remaining images in the second category are mapped;selecting a second feature on which to generate a feature map; andsplitting the first category of images into the first subcategory andthe second subcategory based on the feature map generated for theselected second feature.
 14. The system of claim 9, wherein splittingthe first category of images into the first subcategory and the secondsubcategory comprises: determining that the second area in the featuremap overlaps with an area in the feature map in which a subset of imagesin the second category are mapped, the area in the feature map in whichthe subset of images in the second category are mapped being a thresholddistance away from an area in the feature map in which remaining imagesin the second category are mapped; modifying one or more aggregationparameters used to generate the feature map; regenerating the featuremap for the one or more features based on the modified aggregationparameters; and splitting the first category of images into the firstsubcategory and the second subcategory based on the regenerated featuremap.
 15. The system of claim 9, wherein retraining the CNN based on atraining data set including images in the first subcategory, images inthe second subcategory, and images in the second category comprises:determining a number of images in each of the first subcategory, thesecond subcategory, and the second category; and balancing images in thefirst subcategory, the second subcategory and the second category suchthat a difference between the number of images in the first subcategory,the number of images in the second subcategory, and the second categoryis within a threshold amount.
 16. The system of claim 9, wherein theoperation further comprises: receiving a set of images to add to thetraining data set; classifying the received set of images into one ofthe first subcategory, the second subcategory, or the second categorybased on a value of the one or more features for each image in thereceived set of images to add to the training data set; and retrainingthe CNN based on the training data set and the received set of images toadd to the training data set.
 17. A computer-readable storage mediumhaving instructions stored thereon which, when executed by a processor,performs an operation for training a machine learning model to classifyimages, the operation comprising: receiving a training data setincluding images in a first category and images in a second category;training a convolutional neural network (CNN) using the receivedtraining data set; generating a feature map from one or more layers ofthe CNN based on one or more features of images in the training dataset; identifying a first area in the feature map including images in thefirst category and a second area in the feature map where images in thefirst category overlap with images in the second category; based on theidentified first and second areas, splitting the first category ofimages into a first subcategory corresponding to the first area in thefeature map and a second subcategory corresponding to the second area inthe feature map; and retraining the CNN based on a training data setincluding images in the first subcategory, images in the secondsubcategory, and images in the second category.
 18. Thecomputer-readable storage medium of claim 17, wherein the operationfurther comprises: deploying the retrained CNN to an image classifier,wherein the image classifier is configured to: receive an image forclassification; classify the image using the retrained CNN into one of aplurality of internal categories, the internal categories including atleast the first subcategory, the second subcategory, and the secondcategory; identify a category for the image based on a mapping betweenthe plurality of internal categories and a plurality of categories; andreturn the identified category as the classification of the image. 19.The computer-readable storage medium of claim 17, wherein splitting thefirst category of images into the first subcategory and the secondsubcategory comprises: determining that the second area in the featuremap overlaps with an area in the feature map in which a subset of imagesin the second category are mapped, the area in the feature map in whichthe subset of images in the second category are mapped being a thresholddistance away from an area in the feature map in which remaining imagesin the second category are mapped; selecting a second feature on whichto generate a feature map; and splitting the first category of imagesinto the first subcategory and the second subcategory based on thefeature map generated for the selected second feature.
 20. Thecomputer-readable storage medium of claim 17, wherein the operationfurther comprises: receiving a set of images to add to the training dataset; classifying the received set of images into one of the firstsubcategory, the second subcategory, or the second category based on avalue of the one or more features for each image in the received set ofimages to add to the training data set; and retraining the CNN based onthe training data set and the received set of images to add to thetraining data set.